Runway AI video gets a big upgrade — 5 prompts to try it out
Runway is my favorite artificial intelligence video generator. It has a flexible range of features, options, and capabilities in addition to an impressively realistic motion and visual style that make it a great all-rounder. With the latest update to Runway, you can now have more control over the way it handles images as the starting point for a video.
Following in the footsteps of Luma Labs Dream Machine and Kling, you can now give Runway Gen-3 Turbo the first and final image of a video. It will then fill in the gaps between them to turn that into 10 seconds of video. In theory, this allows for much greater degrees of control. In practice, you can end up with some strange outcomes.
You can either leave the text prompt completely blank and allow the AI to work out how to handle the transition from first to final frame, or you can create your own description of how the motion should work.
Putting Runway Gen-3 to the test
To get a better idea of just how well this transition works I created 15 sets of image pairs in Midjourney. The first and last image made for a potentially logical transition. The idea being that one would start the video, the other end it.
I then worked through the Runway prompting guide to come up with a transition description that should work for each of the image pairs. I eventually cut it down to five pairs after some Midjourney images didn’t work or, in the case of an attempted transition from T-Rex to Seagull, the transition didn’t work.
Here’s the final five, including the image prompts and transition prompts if you want to try each of them out for yourself. If you leave a comment, I might share the T-rex pairing.
1. Urban to Nature Transformation
First up, let’s see if we can move from the city to the forest. All the best cities have large swathes of forests, woods or parklands right there in the middle so why not see if we can get AI to move seamlessly between the two scenes?
Start image prompt: “A bustling city street with towering skyscrapers, cars, and pedestrians. The sky is bright, with modern glass buildings reflecting sunlight.”
End image prompt: “A peaceful forest scene with tall trees, lush greenery, and a serene river. The sunlight filters softly through the leaves.”
Transition prompt: “Slowly tracking shot as the bustling city street gradually fades into a green forest, the skyscrapers in the distance becoming partially obscured by thick foliage. The noise of the city grows quieter as nature overtakes the scene, and birds begin to chirp in place of cars.”
It was a bit of a mixed result, getting some of the transition but weirdly having trees appear mid-view or completely cutting off the heads of a statue.
2. Day to Night in a Countryside
Next, we’re going to head into the farmbelt but more to see if the AI can handle a day-to-night transition than for any specific crop.
Start image prompt: “A vibrant field at sunrise, with golden wheat swaying in the breeze, a barn in the distance, and soft sunlight illuminating the sky.”
End image prompt: “A quiet, starry night over the same field, the barn in silhouette, with a crescent moon and stars glowing above.”
Transition prompt: “Timelapse effect as the vibrant daylight dims and the golden wheat fields slowly darken. The sunlight fades behind the barn, stars begin to twinkle in the sky, and the crescent moon emerges, casting a gentle glow over the now-quiet field.”
This was one of my favorite transitions, handling the move smoothly despite the images not being perfectly aligned.
3. Old Ruins to a Futuristic City
The image I generated with Midjourney was a bit more literal than I expected but it was a great look so I kept it. I liked the darker vibe of the skyscrapers but would they merge one from another?
Start image prompt: “Ancient stone ruins overgrown with vines, with cracks in the structures and nature reclaiming the space, surrounded by a dense jungle.”
End image prompt: “A futuristic city skyline with sleek metallic skyscrapers, neon lights glowing against the dark sky, and advanced technology humming in the air.”
Transition prompt: “A rising pan shot as the ancient ruins slowly merge with futuristic skyscrapers. The vines recede as neon lights begin to pulse, and the stone structures are absorbed into gleaming metal towers. The jungle turns into a high-tech cityscape, complete with glowing holograms.”
I was genuinely surprised by the outcome. It failed in that it didn’t “slowly merge,” as in growing skyscrapers out of the ruins, but the transition effect was really impressive. I enjoyed the fact that part of the ruins remained, and we moved past it.
4. Desert to Oasis
What’s better than an oasis in the desert? An Oasis playing “Wonderwall!” Here, we’re having the AI transition from desert scene to stunning oasis.
Start image prompt: “A vast desert with rolling sand dunes, the sun blazing overhead, and no sign of life except for a distant mirage shimmering in the heat.”
End image prompt: “A lush oasis with palm trees, a sparkling clear pond, and a small waterfall cascading into the water, surrounded by greenery.”
Transition prompt: “A continuous pan across the sand dunes, as the desert slowly transforms into an oasis. The sun fades slightly, and green palm trees gradually appear around a shimmering pond, which was once just a mirage on the horizon. The barren sand gives way to lush greenery.”
The outcome was different to what I was expecting. I gave it a fairly broad image of a desert so was hoping for a sideways pan, but I love the final result.
5. Horse and Cart to Car
Finally, we’ll ask the AI to go from an image of a rural cart and buggy to one of a car, both on a country road. Hoping for a seamless transition between them as they drive.
Start image prompt: “A traditional horse and cart moving along a dirt road, the sun shining over an old countryside landscape with wooden fences and a small barn in the distance.”
End image prompt: “A sleek modern car driving along a paved road, the countryside now transformed with smooth tarmac and telephone poles lining the road, with a gas station visible in the distance.”
Transition prompt: “A forward-tracking shot as the horse and cart gradually fades, the dirt road morphing into a paved one, and the cart evolving into a car. The countryside becomes more modern, with fences turning into guardrails and the barn replaced by newer structures.”
I nearly gave up on this one. It took five attempts until I got the transition I was hoping for. There were a couple that came close, including one that saw a random truck drive up behind the buggy, transition to the car view and turn back to the buggy. I got what I wanted in the end.
More from Tom’s Guide
Source link
/cdn.vox-cdn.com/uploads/chorus_asset/file/23641763/acastro_220614_5290_0001.jpg)
